Towards Language-Guided Visual Recognition via Dynamic Convolutions

In this paper, we are committed to establishing a unified and end-to-end multi-modal network via exploring language-guided visual recognition. To approach target, first propose novel convolution module called Language-guided Dynamic Convolution (LaConv). Its kernels dynamically generated based on natural language information, which can help extract differentiated features for different examples. Based the LaConv module, further build fully language-driven network, termed as LaConvNet, unify recognition reasoning in one forward structure. validate conduct extensive experiments seven benchmark datasets of three vision-and-language tasks, i.e., question answering, referring expression comprehension segmentation. The experimental results not only show competitive or better performance LaConvNet against existing networks, but also witness merits an structure, including compact low computational cost high generalization ability. Our source code is released SimREC project: https://github.com/luogen1996/LaConvNet .